A generalization gap estimation for overparameterized models via the Langevin functional variance

This paper discusses the estimation of generalization gap, difference between performance and training performance, for overparameterized models including neural networks. We first show that a functional variance, key concept in defining widely-applicable information criterion, characterizes gap even settings where conventional theory cannot be applied. As computational cost variance is expensive models, we propose an efficient approximation function Langevin (Langevin FV). method leverages only 1st-order gradient squared loss function, without referencing 2nd-order gradient; this ensures computation implementation consistent with gradient-based optimization algorithms. demonstrate FV numerically by estimating gaps linear regression non-linear network containing more than thousand parameters therein.